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Abstract 

Electronic Flight Bags (EFB) are 
increasingly integral to flight deck information 
management. A piloted simulation study was 
conducted at NASA Langley Research Center, 
one aspect of which was to evaluate the usability 
and acceptability of EFBs for viewing and 
managing Notices to Airmen (NOTAMs) and 
data linked aeronautical information services 
(AIS) and meteorological information (MET) . 

The study simulated approaches and 
landings at Memphis International Airport 
(KMEM) using various flight scenarios and 
weather conditions. Ten two-pilot commercial 
airline crews participated, utilizing the Cockpit 
Motion Facility’s Research Flight Deck 
(CMF/RFD) simulator. Each crew completed 
approximately two dozen flights over a two day 
period. 

Two EFBs were installed, one for each 
pilot. Study data were collected in the form of 
questionnaire/interview responses, audio/video 
recordings, oculometer recordings, and 
aircraft/system state data. 

Preliminary usability results are reported 
primarily based on pilot interviews and 
responses to questions focused on ease of 
learning, ease of use, usefulness, satisfaction, 
and acceptability. Analysis of the data from the 
other objective measures (e.g., oculometer) is 
ongoing and will be reported in a future 
publication. 

This paper covers how the EFB 
functionality was set up for the study; the 
NOTAM, AIS/MET data link, and weather 
messages that were presented; questionnaire 
results; selected pilot observations; and 
conclusions. 


Acronyms 

ACARS 

Aircraft Communications 

AGL 

Addressing and Reporting 
System 

Above Ground Level 

A1RMET 

Airmen’s Meteorological 

AIS 

Information 

Aeronautical Information 

AOC 

Services 

Airline Operations Center 

ATC 

Air Traffic Control 

CMF/RFD 

Cockpit Motion Facility’s 

COTS 

Research Flight Deck 
Commercial Off the Shelf 

DA 

Decision Altitude 

Data Comm 

Data Communications 

D-ATIS 

Digital Automatic Terminal 

EFB 

Information Service 
Electronic Flight Bag 

FAA 

Federal Aviation Administration 

HAT 

Height Above Touchdown 

1AP 

Initial Approach Procedure 

1LS 

Instrument Landing System 

KMEM 

Memphis International Airport 

LOC 

Localizer 

MET 

Meteorological 

METAR 

Aviation Routine Weather 

MSL 

Report 

Mean Sea Level 

NEXRAD 

Next-Generation Radar 

NM 

Nautical Mile 

NOTAM 

Notice to Airmen 

PIREP 

Pilot report 

PFD 

Primary Flight Display 

RNAV 

Area Navigation 

SID 

Standard Instrument Departure 

SIGMET 

Significant Meteorological 

TAF 

Information 

Terminal Aerodrome Forecast 


Background 

EFBs are used in current airline operations, 
but on a relatively small scale, and with 
somewhat limited functionality with respect to 
their potential. EFBs can display a variety of 
aviation data. Some of this information has been 
traditionally provided in paper form or based on 
data provided to the crew by an airline’s flight 
dispatch function. Operators and flight crews are 
transitioning from paper products in a traditional 
flight bag to an electronic format. 

EFBs may be portable (Class 1), attached to 
a mounting device (Class 2), or built into the 
aircraft (Class 3) [1]. 

An industry survey of Class 1, 2, and 3 
EFB capabilities was conducted in 2010 [2], The 
survey was conducted among: 

• 18 systems manufacturers who 

provide both EFB hardware and 

software 

• 5 software manufacturers who offer 

integrated and customizable 
software without hardware 

• 53 software manufacturers who 

provide commercial off-the-shelf 
(COTS) software not integrated or 
tailored for a particular EFB. 

Table 1 shows the range of EFB 
capabilities and applications that the survey 
found were supported. The capabilities are listed 
in aggregate descending order of prevalence. 

The survey does not explicitly break out the 
capability to access NOT AMs or A1S/MET 
information, however much of this type of 
information is embedded in categories Flight 
Planning and Weather. 

The FAA provides guidance on the 
operational use of EFBs through [1]. It provides 
some key EFB design recommendations: 

• EFBs may display data li nk ed A1S 
and MET information for advisory 
use to support strategic decision 
making and to enhance situational 
awareness. 


Table 1. EFB Capabilities 


Capability 

System 

Mfrs, 

% 

Int’ed 

SW, 

% 

COTS 

SW, 

% 

Electronic charts 

100 

100 

4 

Flight planning 

78 

100 

35 

Flight performance 
calculations 

100 

100 

25 

Electronic docs 

94 

100 

25 

Weather 

39 

100 

42 

En route moving 
map 

83 

60 

23 

Logbook 

67 

60 

19 

Electronic 

checklists 

89 

80 

6 

Surface moving 
map 

89 

60 

6 

Video surveillance 

78 

80 

2 

Terrain display 

56 

0 

19 

GPS/Nav display 

61 

0 

4 

Data link 

50 

40 

4 

Voice data comm 

44 

20 

0 

Enhanced vision 

44 

0 

2 

Traffic surveillance 

33 

0 

6 

Synthetic vision 

22 

0 

4 


• Data linked graphical weather (e.g. 

NEXRAD) may be used for 
strategic/flight planning purposes. 

• The EFB user interface should be 

consistent and intuitive across the 
EFB and its various applications. 

• Electronic aeronautical charts 

should provide a level of 
information comparable to paper 
charts. 

• Requirements for approval of airport 

moving map displays (AMMD) for 
EFB’s are specified in [3]. 
AMMD’s help flight crews orient 
themselves on the airport surface 
and improve positional awareness 
during taxi operations. 

• The EFB design should minimize 

crew workload and head-down time 
to avoid multi-step tasks during 
takeoff, landing, and other critical 
phases of flight. 


Study Set Up 

The assessment reported here was part of a 
concurrent effort to study how pilot decision- 
making in the presence of complexity is 
influenced by information management, content, 
and quality. The results of that larger effort are 
reported elsewhere [4]. The EFB aspect of the 
study sought to assess the usability and 
acceptability of EFBs for providing crews with 
NOTAMs and data linked A1S/MET 
information. Some information was presented as 
text only, some graphical only, and some in both 
text and graphical forms. 

The experiment was hosted by the 
CMF/RFD over the period from March 27 to 
May 3, 2013. The physical structure is shown in 
Figure 1. For this study, the EFBs were 
repositioned more inboard and lower than 
shown. Also, the right-side head-up display is 
not shown. The CMF/RFD simulator has 
commonalities with the Boeing 757 and 767 
(aerodynamic model, handling qualities, flight 
management system, control display unit, 
automation mode control panel, center aisle 
stand and throttle quadrant, and overhead panel), 
the Boeing 787 (four 17-inch liquid crystal 
displays, dual FIUDs, and dual EFBs), and 
Airbus aircraft (dual side-sticks). 

Over the course of the experiment, 1 0 two- 
pilot airline crews participated. Each crew 
consisted of a Captain and First Officer from the 
same airline, with four airlines represented. 


Each crew completed a set of flights into 
Memphis International Airport (KMEM). All 
flights began at 10,000 ft MSL, on one of four 
RNAV-based arrival and approach paths, and at 
an airspeed of 250 kn ots. Flights ended 
nominally after landing and runway exit; 
however, in many cases the crew decided to go- 
around. Flight times ranged from 8 to 15 
minutes. 

Three unique weather conditions were used 
during the flights. For the first, visibility was 3 
NM and the cloud ceiling was 1000 ft. Winds 
were light, and there was no turbulence or rain. 
For the second and third conditions, storm 
models were used based on two events that 
occurred in 2011, and for which KMEM had 
significant weather-induced delays. The two 
storms were different in intensity and variability. 
These three models drove the motion-platform 
(e.g. winds and turbulence effects) and for this 
part of the study, determined what weather 
information should be provided on the EFBs. 
For more on how the weather environment was 
created and used to drive the flight simulation 
see [5]. 

Data communications (Data Comm) was 
implemented as follows. Messages from ATC 
were displayed on the PFD and the EFBs. The 
EFBs were used for reviewing and accepting or 
rejecting the ATC messages. Each incoming 
message was announced via a chime. NOTAMs 
and weather information were provided as a pre- 
flight load on the EFB (e.g. to support pre-flight 
briefings) and via data link in flight. 



Figure 1. Cockpit Motion Facility’s Research Flight Deck 


Data from the study were collected in 
several forms including interview and 
questionnaire responses, audio/video recordings, 
pilot input (e.g. button presses), head and eye 
tracking, and pilot and researcher observations. 
Short questionnaires were completed by each 
pilot following each flight. 

EFB Functionality for Study 

It was the intent of this study to adhere to 
EFB-related current standards and practices and 
to mimic and extend current products and 
capabilities. 

A baseline set of EFB functions was 
implemented consistent with current state-of- 
the -practice. From the main menu of the EFB, 
seven categories of information were available: 

• Communications 

• Weather 

• NOTAMs 

• Charts 

• Airport Moving Map 

• Checklists 

• D-ATIS 

Two Class 3 EFBs were installed, left and 
right, to allow each pilot to access information 
independently. (FAA regulations require at least 
two operational EFBs in order to remove paper 
aeronautical charts, checklists, or required data 
from the flight deck. [1]) Data link messages 
were echoed on both units. This enabled 
functions such as acknowledging messages to 
only have to be done from one EFB. 

On receipt of a data link message, either 
from ATC or via the A1S/MET service, one of 
the pilots was required to review the message, 
discuss it with the other pilot, then reply using 
the EFB (e.g. via a “wilco” message). Prior to 
the start of each flight, the crew was required to 
review and discuss current NOTAMs and 
weather information stored in the EFB. The 
crew was required to use the charts and 


checklists in the EFB per their typical airline 
procedures. 

No paper reference material was available 
during the experiment, other than a few pages 
from the operating manual and quick reference 
handbook that were occasionally needed to 
support emergency checklist items. 

For this experiment, the A1S/MET data li nk 
functionality was consistent with the current 
aviation industry concept of use for A1S/MET 
data link services [6]. Depending on the scenario 
flown, the AIS/MET messages received during a 
flight were drawn from the following: 

• Temporary obstruction (text and 

graphic) 

• Operationally significant NOTAMs 

(text and graphic) 

• Changes to instrument approach 

procedures (text and graphic) 

• Runway braking action reports (text 

only, no graphic) 

• Runway winds (text only, no 

graphic) 

• Pilot reports (text only, no graphic) 

The airport moving map functions and 
displays were implemented as in [7], with some 
changes and additional features. The moving 
map depicted locations of runway and taxiway 
edges, painted markings, and buildings, with 
overlays of aircraft position, traffic positions, 
and the ATC-provided taxi route. The taxi route 
was provided via Data Comm. The moving map 
was integrated with the NOTAM system such 
that geo-referenced NOTAMs were displayed 
graphically (e.g. closed runways). 

The EFB layout is shown in Figure 2 
(inactive buttons are grayed-out). The main 
menu page was visible at startup and accessible 
at any time by pressing the MENU button at the 
top of the tablet. 

The usable screen size was 6.2” wide by 
8.3” high and the resolution was 1024 x 768 
(XGA). 



Right EFB 


Pilots access 
sub-menus by 
pressing 
touchscreen 
label or the 
associated 
bezel button 



• COMM -To review and acknowledge 
data link messages from ATC and the 
AIS/MET service 

• WEATHER -To review MET reports 
(METARs), Terminal Area Forecasts 
(TAFs), Airmen's MET reports 
(AIRMETs), Pilot reports (PIREPs), Winds 
aloft, and composite ground-based 
radar images 

• NOTAMs - To review NOTAMs in text 
and graphic forms (during pre-flight or 
when received in-flight via the AIS/MET 
service) 

• AIRPORT MOVING MAP - To view 
airport features with position overlay, 
traffic, and taxi route 

• CHARTS (Arrival, Approach, Airport) 

• CHECKLISTS (Descent, Landing, Taxi) 

• ATIS (Automatic Terminal Info Service) 


Figure 2. EFB Functionality 


Comm Pages 

Upon receipt of a data link message, 
“COMM MSG” is displayed in green in the 
upper right portion of the EFB. A chime is also 
provided. The “COMM MSG” indication 
remained until all data link messages were 
responded to. On the COMM page the text of 
data link messages is green until a response is 
sent, at which time, the text turns white. See 
Figure 3. 

AIS/MET messages are displayed in order 
of receipt, with most recent at the top. A “(G)” 
in front of the message indicates an associated 
graphic is available. Touching the (G) message 
or pressing the associated bezel button brings up 
the message in graphical form. Acknowledging 
a message changes the text to white. 
Unacknowledged messages remain green. 

The most recent ATC data link message is 
displayed in the upper portion of the page. 



Figure 3. Communications Page 

























Weather Pages 

The following weather products were available 
on the EFB for this study: 

• METAR (text) 

• TAF (text) 

• D-ATIS (text) 

• Winds (graphic) 

• NEXRAD (graphic) 

• A1RMET Sierra (text plus graphic) 

• A1RMET Tango (text plus graphic) 

• A1RMET Zulu (text plus graphic) 

• PIREPs (text plus graphic) 

All were provided in a form similar to that 
from current commercial EFB solution providers 
(e.g., ForeFlight, Garmin Pilot, WingX). The 
information content for these depended on 
which of the three weather models was used in a 
given scenario. For a detailed explanation of 
these weather products see [8]. For more 
information on how Winds and NEXRAD 
graphics were created based on the weather 
models see [5]. 

NOTAMs Pages 

As with the A1S/MET messages, a “(G)” in 
front of the text indicates that an associated 
graphic is available. Each text NOTAM is 
associated with a button on the left side of the 
EFB. Pressing it displays the corresponding 
graphical NOTAM. See Figure 4 for an example 
NOTAMs page. 

Airport Moving Map Page 

The airport moving map page is described 
in [4] and [7], 

Charts Pages 

The CFIARTS pages provide the published 
arrivals, approaches, departures, and airport 
diagram for KMEM. 

AIS/MET Messages 

A1S information is defined in [9]. The A1S 
messages used in this study were developed in 
accordance with [6], which represents the 
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Figure 4. NOTAMs Page 

aviation industry view on how A1S and MET 
data link services woidd support flight 
operations in the future. The following A1S 
messages (text and graphic forms) were used in 
the study: 

"KMEM Rwy 18R/36L north 500 ft closed" 

"Change to IAP for ILS or LOC Rwy 36C. DA 
551 ft. HAT 230 ft. RVR 2400 ft" 

"MEM Temporary Obstruction. Crane 0.15 
NM west of approach to 36L. Height 401 
MSL (60 AGL) at 350214N, 892921W" 

The following MET messages (text only) 
were used: 

"KMEM PIREP: Low level wind shear, +/- 10 
knots, surface to 1000 ft during approach, 
Runway 18C" 

"Runway 18C braking condition poor" 

"Runway 36C arrival, current threshold 
wind from 180 degrees at 10 knots" 

Figure 5 shows the graphic that was used for 
"KMEM Rwy 18R/36L north 500 ft closed." 
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Figure 5. Runway 18R/36L N 500 ft Closed 


NOTAMs 

About two dozen NOTAMs were 
loaded into the EFB for each scenario. The 
quantity and type NOTAMs used in the study 
were consistent with historical KMEM 
NOTAMs. Those used in the study were 
developed from published NOTAMs for KMEM 
available from the US NOT AM system at [10]. 

These were tracked over time to develop a 
representative sample. The NOTAMs were not 
filtered or tailored to the specific flight and thus 
many were not pertinent. This was done for 
consistency with the current NOTAM 
environment. Traditional text format NOTAMs 
used in the study included: 

• Change to departure procedures 

• Change to arrival procedures 

• Airport equipment out of service 

• Lights out of service 


• Birds in vicinity of airport 


Some graphical NOTAMs were based on 
the work of SAE G-10A, which at the time of 
this study was developing [11], These graphical 
NOTAMs were: 

• Runway closed 

• Runway shortened 

• Taxiway closed 

• Glide path out of service 

• Change to instrument approach 
procedure 

Other graphical NOTAMs used were: 

• Special Use Airspace 

• Airborne laser operations 

• Temporary Flight Restriction 

All graphical NOTAMs had associated text 
in the traditional NOTAM format available via 
the SHOW/HIDE TEXT button. See Figure 6 
for an example. 


Results 

Preliminary findings are presented here 
regarding usability and acceptability of the EFBs 
for reviewing NOTAMs and A1S/MET data li nk 
messages. Selected pilot comments are also 
provided. Data analysis continues and a more 
comprehensive set of results will be presented in 
a future publication. This will include, for 
example, an analysis of oculometer data to 
determine the amount attentional resources used 
by the pilots to interact with EFBs during the 
approaches. 

For the 1 0 commercial crews (n = 20 pilots) 
that participated, response percentages when 
asked for their current method of accessing 
NOTAMs during pre flight were: 

• Paper (81.8%) 

• Paper and EFB (9.1%) 

• Paper, EFB/tablet, ACARS (4.5%) 

• EFB/tablet (4.5%) 
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Figure 6. Runway 9/27 Closed Until Further 
Notice (Graphic with Text Overlay) 

Response percentages as to the current method 
of accessing NOTAMs during a flight were: 

• ACARS (40.9%) 

• Paper and ACARS (40.9%) 

• Paper (13.6%) 

• Paper and tablet (4.5%) 

The use of EFBs by the crews to access 
NOTAMs was not a prevalent practice. 

Usability 

Usability questionnaires were developed 
based on [12] and administered individually to 
each pilot following each flight. Responses were 
provided using a Likert scale of 1 (Strongly 
Disagree) to 7 (Strongly Agree). Assessments 
were limited to the usability of EFBs for 
viewing and managing NOTAMs and data 
linked A1S/MET messages. Four categories of 
usability were assessed: 

• Usefulness 


• Ease of use 

• Ease of learning 

• Satisfaction 

Pilots rated multiple subfactors for each of 
these categories. Results are summarized and 
presented in Figures 7 through 10. Pilots rated 
usability with respect to pre -flight use and in- 
flight use. The percentage of responses for a 
given rating is shown to the right of the bars in 
the figures. For each category, three statistical 
measures are shown: median (x), mode (Mo), 
and number of responses (n). The number of 
responses varies, mainly because the number of 
subfactors for each of the four categories was 
not equal, but also because subfactor ratings 
were occasionally left blank. 

As indicated by Figure 7, Usefulness was 
rated high by a large preponderance of the pilots 
for both preflight and in-flight. The median 
Usefulness rating was 6 or 7 in 80 % of the 
preflight responses (46% + 34%) and in 79% of 
the in-flight ratings (41% + 38%). Also, there is 
no significant difference between preflight and 
inflight use. For both preflight and in-flight the 
median response was 6 and the mode was 7 and 
over 90% of the responses were 5 or above. 

The results for Ease of Use (Figure 8) are 
more mixed. For preflight the median response 
was 6 and the mode was 7. For in-flight the 
median response was 6 but the mode fell to 5. 
This is due in part to the fact that reaching 
certain pages on the EFB required multiple 
button presses. 

Ease of Learning was very highly rated, 
both for preflight and in-flight use (Figure 9). 
Most pilots had little difficulty learning to use 
the EFB and quickly becoming proficient. 

Although still high, Satisfaction was the 
lowest rated usability component (Figure 10). 
This reflects some of the dislikes listed in the 
Pilot Comments section below and is related to 
aspects of page navigation, distracting messages, 
the message acknowledgement process, and the 
lack of message filtering or tailoring to the 
specific flight (although that is consistent with 
today’s NOTAM environment). All of these 
issues can be resolved during the design 
refinement process. 




Rating Rating 


□ Preflight (x = 6, Mo = 7, n = 144) 
■ In-flight (x = 6, Mo = 7, n = 162) 



Percentage of Responses 


□ Pre-flight (x = 6, Mo = 7, n = 92) 
■ In-flight (x = 6, Mo = 7, n = 96) 



Percentage of Responses 


Figure 7. Usefulness Ratings 


Figure 9. Ease of Learning Ratings 


□ Preflight (x = 6, Mo = 7, n = 184) 
■ In-flight (x = 5, Mo = 5, n = 206) 



□ Pre-flight (x = 5, Mo = 5, n = 69) 
■ In-flight (x = 5, Mo = 6, n = 73) 
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Figure 8. Ease of Use Ratings 


Figure 10. Satisfaction Ratings 


Acceptability 

Acceptability was assessed through pilot 
ratings of the eight factors listed in Table 11. 
Pilots rated the factors on a Likert scale of 1 
(Completely Disagree) to 7 (Completely Agree). 
Summary results are given in Table 1 1 for Part 1 
and Part 2 separately. In Part 1 use of the EFB 
was introduced to the pilots and there were 
fewer complexities involved in the operations 


(e.g. no traffic). In Part 2, complexity was 
increased by introducing off-nominal events, 
adding traffic, and requiring the pilots to 
maintain prescribed spacing with lead aircraft 
[4], The number of responses (n) is greater for 
Part 2 than for Part 1 because more scenarios 
were flown in Part 2. 


Table 11. Acceptability Factors and Ratings 


# 

Factor 

Median 

Mode 

n 

Part 

1 

Part 

2 

Part 

1 

Part 

2 

Part 

1 

Part 

2 

1 

1 was aware of the AIS/MET data link messages within 
an appropriate timeframe 

6 

6 

6 

6 

67 

219 

2 

1 was surprised to notice that an AIS/MET data link 
message was sent and wondered how long ago it has 
been sent 

2 

1 

2 

2 

67 

222 

3 

1 was aware of all relevant NOTAMs throughout the 
scenario 

6 

6 

6 

6 

67 

226 

4 

Use of the EFB for awareness of the NOTAMs and 
AIS/MET information in this scenario is acceptable 

6 

6 

6 

6 

67 

225 

5 

The NOTAM and AIS/MET information 1 received are 
operationally realistic 

6 

6 

6 

6 

66 

225 

6 

Time required for EFB tasks did not detract from 
having appropriate situation awareness for other 
aspects of flight 

5 

6 

6 

6 

67 

219 

7 

1 received AIS/MET data link messages when in the 
process of completing other critical tasks 

5 

6 

6 

6 

65 

223 

8 

At no point in this scenario did 1 feel the NOTAM or 
AIS/MET information conflicted with other available 
information 

6 

6 

6 

6 

66 

219 


These ratings indicate that the EFBs were 
generally acceptable to pilots for reviewing and 
managing NOTAMs and A1S/MET data link 
messages. Some improvement in median rating is 
seen going from Part 1 to Part 2, despite the added 
operational complexity. Median ratings improved for 
factors 2 and 6. Although the complexity of the 
scenarios significantly increased in Part 2, the pilots 
were less often surprised to notice an A1S/MET 


message and felt more strongly that the time required 
for EFB tasks did not detract from situational 
awareness for other aspects of the flight. These 
improved ratings can be largely attributed to the 
benefit of familiarization with the EFBs during Part 1 
and a resultant increase in proficiency with using the 
EFB during Part 2. Only factor 7 was rated less 
acceptable in Part 2. This is not surprising since by 
design the pilots were occupied with more critical 


tasks in Part 2. The mode for each factor indicates 
good acceptability and remained unchanged from 
Part 1 to Part 2. 

Pilot Comments 

The following list provides selected comments 
and suggestions from the pilots. As with any such 
study, pilots sometimes present inconsistent 

viewpoints; however, below the salient issues are 
presented. 

Likes: 

1. The EFB as a message listing and viewing 
concept was well received. It minimized looking 
at different places for information. 

2. A graphic display is worth 1000 words. 
Graphical NOTAMs and AIS/MET messages 
offer very quick understanding versus reading 
text and are easy to follow. They cut through 
reams of text, and save time for crews by 
highlighting important information. 

3. Using the charts to display the ILS glide path out 
of service NOT AM and change to approach 
procedure NOT AM was very effective: easy to 
identify, easy to follow, easy to apply. 

4. The aural notice of a new AIS/MET message was 
helpful, especially during high workload when a 
message might otherwise be missed. 

5. AIS/MET messages alert the crew to safety 
issues that may have been missed during 
preflight or from new conditions developing. 
Also, messages relating to airport conditions are 
valuable as backup to tower controllers. 
Redundancy can be good. 

Dislikes: 

1. A big frustration was that several key strokes 
were sometimes required to get back to the page 
needed, especially the approach plate and key 
charts. Pilots want to be able to get back quickly, 
without using two or three keystrokes. 

2. Only NOTAMs and AIS/MET messages that are 
significant to the flight should be shown. 
Irrelevant messages should be filtered out. If a 
message did not affect the flight, most pilots did 
not want to know about it. For example, “I don’t 
need to know that last 500 ft of runway 36L is 


closed if I’m landing on runway 36C.” However, 
significant weather changes would always be 
good to know. If something just changed from 
the last ATIS, it should be data li nk ed up. 

3. Distracting messages that come during a critical 
phase of flight (e.g., at low altitude during 
approach) are not good. Messages should be 
minimized to urgent issues like windshear, poor 
braking action, or weather minima change. Pilots 
need to know what to expect on approach by top 
of descent, not inbound at final approach fix or 
later. 

4. The acknowledge process needs to be quicker 
and not require losing the displayed chart. Some 
messages should maybe require no 
acknowledgement or just a click to Accept, 
Reject, or Cancel. Having to cycle to a different 
page and back is cumbersome and wastes 
valuable time in a critical phase of flight. 
Available responses could be displayed with the 
comm message when selected. Once the response 
is made, the display should go back to prior page. 

5. When the graphical taxi route is shown crossing a 
closed runway, there should be a visual 
indication that it is OK to cross the closed 
runway. Otherwise it creates confusion as to 
whether the aircraft is cleared to cross. 


Suggestions to improve EFB functionality: 

Note: These suggestions pertain to the EFB as 
configured for this study. 

1 . Set up the EFB with the icons on the screen and a 
“home” button to show the last viewed pages for 
quick selection. 

2. Add pinch and stretch capability for zooming 
(two-finger touchscreen capability). 

3. Make the NOTAM text change color after the 
graphic is accessed to show that the NOTAM has 
already been viewed. 

4. The depiction of closed runways and taxiways 
with X’s was good, but consider flashing also to 
increase attention. 

5. Tie the FMS and EFB together so that if an 
arrival and approach is selected on the FMS it 


comes up on the EFB without having to look for 
it. 

6. Add the capability to create pilot-defined buttons 
to get the needed information with one button 
press without going back through the menus (i.e. 
hot-keys). 

7. Add a hot key or auto return to the approach 
plate. 

8. Add a hot key for the missed approach chart. 

9. Add an alert light or flashing button indicator for 
new A1S/MET messages. 

10. Add automated notification of ATIS updates. 

Next Steps 

The next steps for this research are to analyze 
the data in greater detail by looking at the impact of 
weather, scenario complexity, and the alternating 
roles (flying/monitoring) of the two pilots in each 
flight. In addition there are other data from this study 
yet to analyze, including the head and eye tracking 
data (e.g. to determine how much attention is being 
taken by EFBs and by which pages and when and 
how they are used) and the audio/video data. There 
were some cases where pilots may have been using 
the EFBs more often than necessary, or desired, 
pulling their attention away from otherwise important 
tasks. Oculometer data along with EFB page 
switching recordings can help to quantify this issue. 

After the results of this study are assimilated, 
future follow-on studies are envisioned to address 
issues that have been identified. Certainly this study 
points to many opportunities to improve the usability 
of future EFBs for viewing and managing NOTAMs 
and AIS/MET data link messages, and to inform 
evolving standards and regulations related to same. 

Concluding Remarks 

In this study commercial airline pilots rated 
EFBs for usability and acceptability for viewing and 
managing NOTAMs and AIS/MET data link 
messages. Most of the pilots had little to no prior 
experience with this concept. However, in general 
they learned quickly, becoming proficient in a 
relatively few number of flights. 


In general the pilots rated the usability and 
acceptability of the EFBs well, but offered 
suggestions for improving the user experience based 
on their operational needs. These suggestions are 
insightful and merit further attention. 

After observing pilot interactions with the EFBs 
over the course of ~240 approaches, several common 
themes emerge: 

1. Procedures for utilizing EFBs for 
reviewing/managing NOTAMs and 
AIS/MET data link messages must be well- 
defined; particularly in a two-pilot crew 
environment. 

2. The value of an EFB as a message listing and 

viewing tool was evident. Using EFBs to 
display and manage NOTAMs and 
AIS/MET information can enhance 
situational awareness and have clear 
operational benefit. 

3 . Pilots tended to leam to use the EFB quickly 

adapted readily to a paperless cockpit 
environment. However, there were cases 
where paper may have benefited (e.g. when 
navigating away from a chart page). 

4. Extensive navigation among the several 
pages and functions increases workload to 
an undesirable and potentially unacceptable 
level. This may lead to an alternate form of 
automation mode confusion (i.e., EFB mode 
confusion); there was one case where the 
two pilots were viewing different charts, 
each believing they were looking at the 
correct one. 

5. The power of graphical presentations to 

convey understanding and awareness 
quickly was clear. Pilots always preferred 
graphical NOTAMs and AIS/MET messages 
to text form. Information should be 
presented graphically wherever possible. 

6. Information retrieval should be quick and 

straightforward. Access to relevant charts in 
a timely manner is essential. An intelligent 
or “smart” EFB may be the best solution for 
some situations (e.g., pop-up info that is new 
and has critical operational importance). 


7. During periods of high workload, especially 

on final approach, distracting messages on 
the EFB should be minimized. Voice was 
the preferred means to pass information 
inside the final approach fix or below a 
given altitude (e.g., 1500 ft AGL). Only 
urgent AIS/MET messages should be sent 
beyond this point. 

8. Pilot interaction with the EFB in critical 
phases of flight should take the fewest 
button pushes possible, preferably one. 

9. NOTAMs and AIS/MET messages should be 

filtered and tailored to the flight. Ideally, 
only relevant information should be 
sent/displayed. 

10. All taxiway and runway closures should be 
available for view on a single diagram. For 
example, issue a single NOTAM (“Airport 
closures, see chart”). 

The study reinforces some time -tested usability 
maxims [13]: 

• The information for the decision should be 
there when the decision is needed. 

• Don't overwhelm the user. 

• Keep it simple. 

• The best journey has the fewest steps. 

Shorten the distance to the goal. 

Ideally, EFB’s should be useful, easy to use, 
easy to learn, and satisfying to use. The results show 
that in this study they generally were, but there is 
room for improvement in future implementations. 
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